Exploring factors that affect performance on introductory programming courses

Researchers have long tried to identify factors that could explain why programming is easier for some than the others or that can be used to predict programming performance. The motivation behind most studies has been identifying students who are at risk to fail and improving passing rates on introductory courses as these have a direct impact on retention rates. Various potential factors have been identified, and these include factors related to students' background, programming behavior or psychological and cognitive characteristics. However, the results have been inconsistent. This thesis replicates some of these previous studies in a new context, and pairwise analyses of various factors and performance are performed. We have data collected from 3 different cohorts of an introductory Java programming course that contains a large number of exercises and where personal assistance is available. In addition, this thesis contributes to the topic by modeling the dependencies between several of these factors. This is done by learning a Bayesian network from the data. We will then evaluate these networks by trying to predict whether students will pass or fail the course. The focus is on factors related to students' background and psychological and cognitive characteristics. No clear predictors were identified in this study. We were able to find weak correlations between some of the factors and programming performance. However, in general, the correlations we found were smaller than in previous studies or nonexistent. In addition, finding just one optimal network that describes the domain is not straight-forward, and the classification rates obtained were poor. Thus, the results suggest that factors related to students' background and psychological and cognitive characteristics that were included in this study are not good predictors of programming performance in our context

Konvoluutioneuroverkot ja Gaussiset prosessit sensoridatan analysoimiseen

Different sensors are constantly collecting information about us and our surroundings, such as pollution levels or heart rates. This results in long sequences of noisy time series observations, often also referred to as signals. This thesis develops machine learning methods for analysing such sensor data. The motivation behind the work is based on three real-world applications. In one, the goal is to improve Wi-Fi networks and recognise devices causing interference from spectral data measured by a spectrum analyser. The second one uses ultrasound signals propagated through different paths to localise objects inside closed containers, such as fouling inside of industrial pipelines. In third, the goal is to model an engine of a car and its emissions. Machine learning builds models of complex systems based on a set of observations. We develop models that are designed for analysing time series data, and we build on existing work on two different models: convolutional neural networks (CNNs) and Gaussian processes (GPs). We show that CNNs are able to automatically recognise useful patterns both in 1D and 2D signal data, even when we use a chaotic cavity to scatter waves randomly in order to increase the acoustic aperture. We show how GPs can be used when the observations can be interpreted as integrals over some region, and how we can introduce a non-negativity constraint in such cases. We also show how Gaussian process state space models can be used to learn long- and short-term effects simultaneously by training the model with different resolutions of the data. The amount of data in our case studies is limited as the datasets have been collected manually using a limited amount of sensors. This adds additional challenges to modeling, and we have used different approaches to cope with limited data. GPs as a model are well suited for small data as they are able to naturally model uncertainties. We also show how a dataset can be collected so that it contains as much information as possible with the limited resources available in cases where we use GPs with integral observations. CNNs in general require large datasets, but we show how we can augment labeled data with unlabeled data by taking advantage of the continuity in sensor data.Erilaiset sensorit keräävät jatkuvasti dataa meistä ja ympäristöstämme, kuten ilmanlaadusta tai ihmisen sykkeestä. Tuloksena on pitkiä aikasarjahavaintoja, joita usein kutsutaan myös signaaleiksi. Tässä työssä kehitetään koneoppimismenetelmiä sensoridatan analysoimiseen. Motivaationa työssä on kolme erilaista käytännön sovellusta. Ensimmäisessä pyritään parantamaan Wi-Fi -verkkojen toimintaa tunnistamalla häiriötä aiheuttavia laitteita spektridatasta. Toisessa käytetään ultraääntä paikallistamaan kohteita suljettujen säiliöden sisällä. Kolmannessa mallinnetaan auton moottoria ja sen päästöjä. Koneoppiminen muodostaa malleja monimutkaisista järjestelmistä havaintojen pohjalta. Tässä työssä kehitetään malleja, jotka sopivat erityisesti aikasarjojen analysointiin. Nämä mallit perustuvat kahteen erilaiseen malliperheeseen: konvoluutioneuroverkkoihin ja Gaussisiin prosesseihin. Työssä kehitetään konvoluutioneuroverkkoja sekä yksi- että kaksiulotteisen signaalidatan analysointiin ja lisäksi osoitetaan, että niiden avulla voidaan tulkita myös signaaleja jotka on hajautettu satunnaisesti mittausalueen kasvattamiseksi. Työssä kehitetään Gaussisia prosesseja tapauksiin, joissa havainnot ovat integraaleja tuntemattoman funktion yli ja yleistetään menetelmä myös tilanteisiin joissa tuntemattoman funktion arvot ovat rajoitettuja, esimerkiksi ei-negativisia. Lisäksi esittelemme tavan, jolla gaussisia prosesseja hyödyntävät tila-avaruusmallit pystyvät oppimaan sekä pitkän että lyhyen aikavälin ilmiöitä käyttämällä opettamiseen datan eri resoluutioita. Työssä käsiteltävissä sovelluksissa datan määrä on verrattain pieni, sillä data on kerätty manuaalisesti vain pienellä määrällä sensoreita. Tässä työssä esitellään myös ratkaisuja pieniin datamääriin liittyviin haasteisiin. Näytämme, miten data voidaan kerätä niin, että se sisältää mahdollisimman paljon informaatiota pienistä resursseista huolimatta, tapauksissa, joissa havainnot vastaavat integraaleja alueiden yli. Konvoluutioneuroverkot tyypillisesti tarvitsevat opettamiseen paljon dataa, mutta työ esittelee miten opettamisessa voidaan täydentää luokiteltua dataa luokittelemattomalla datalla hyödyntämällä sensoridatan aikajatkuvuutta

Semi-supervised Convolutional Neural Networks for Identifying Wi-Fi Interference Sources

We present a convolutional neural network for identifying radio frequency devices from signal data, in order to detect possible interference sources for wireless local area networks. Collecting training data for this problem is particularly challenging due to a high number of possible interfering devices, difficulty in obtaining precise timings, and the need to measure the devices in varying conditions. To overcome this challenge we focus on semi-supervised learning, aiming to minimize the need for reliable training samples while utilizing larger amounts of unsupervised labels to improve the accuracy. In particular, we propose a novel structured extension of the pseudo-label technique to take advantage of temporal continuity in the data and show that already a few seconds of training data for each device is sufficient for highly accurate recognition.Peer reviewe

Automatic Inference of Programming Performance and Experience from Typing Patterns

Studies on retention and success in introductory programming course have suggested that previous programming experience contributes to students' course outcomes. If such background information could be automatically distilled from students' working process, additional guidance and support mechanisms could be provided even to those, who do not wish to disclose such information. In this study, we explore methods for automatically distinguishing novice programmers from more experienced programmers using fine-grained source code snapshot data. We approach the issue by partially replicating a previous study that used students' keystroke latencies as a proxy to introductory programming course outcomes, and follow this by an exploration of machine learning methods to separate those students with little to no previous programming experience from those with more experience. Our results confirm that students' keystroke latencies can be used as a metric for measuring course outcomes. At the same time, our results show that students programming experience can be identified to some extent from keystroke latency data, which means that such data has potential as a source of information for customizing the students' learning experience.Peer reviewe

Non-linearities in Gaussian processes with integral observations

Gaussian processes (GP) can be used for inferring latent continuous functions also based on aggregate observations corresponding to integrals of the function, for example to learn daily rate of new infections in a population based on cumulative observations collected only weekly. We extend these approaches to cases where the observations correspond to aggregates of arbitrary non-linear transformations of a GP. Such models are needed, for example, when the latent function of interest is known to be non-negative or bounded. We present a solution based on Markov chain Monte Carlo with numerical integration for aggregation, and demonstrate it in binned Poisson regression and in non-invasive detection of fouling using ultrasound waves.Peer reviewe

Transfer-Learning Methods in Programming Course Outcome Prediction

The computing education research literature contains a wide variety of methods that can be used to identify students who are either at risk of failing their studies or who could benefit from additional challenges. Many of these are based on machine-learning models that learn to make predictions based on previously observed data. However, in educational contexts, differences between courses set huge challenges for the generalizability of these methods. For example, traditional machine-learning methods assume identical distribution in all data—in our terms, traditional machine-learning methods assume that all teaching contexts are alike. In practice, data collected from different courses can be very different as a variety of factors may change, including grading, materials, teaching approach, and the students. Transfer-learning methodologies have been created to address this challenge. They relax the strict assumption of identical distribution for training and test data. Some similarity between the contexts is still needed for efficient learning. In this work, we review the concept of transfer learning especially for the purpose of predicting the outcome of an introductory programming course and contrast the results with those from traditional machine-learning methods. The methods are evaluated using data collected in situ from two separate introductory programming courses. We empirically show that transfer-learning methods are able to improve the predictions, especially in cases with limited amount of training data, for example, when making early predictions for a new context. The difference in predictive power is, however, rather subtle, and traditional machine-learning models can be sufficiently accurate assuming the contexts are closely related and the features describing the student activity are carefully chosen to be insensitive to the fine differences.Peer reviewe

Typing Patterns and Authentication in Practical Programming Exams

In traditional programming courses, students have usually been at least partly graded using pen and paper exams. One of the problems related to such exams is that they only partially connect to the practice conducted within such courses. Testing students in a more practical environment has been constrained due to the limited resources that are needed, for example, for authentication. In this work, we study whether students in a programming course can be identified in an exam setting based solely on their typing patterns. We replicate an earlier study that indicated that keystroke analysis can be used for identifying programmers. Then, we examine how a controlled machine examination setting affects the identification accuracy, i.e. if students can be identified reliably in a machine exam based on typing profiles built with data from students' programming assignments from a course. Finally, we investigate the identification accuracy in an uncontrolled machine exam, where students can complete the exam at any time using any computer they want. Our results indicate that even though the identification accuracy deteriorates when identifying students in an exam, the accuracy is high enough to reliably identify students if the identification is not required to be exact, but top k closest matches are regarded as correct.Peer reviewe

Identification of Programmers from Typing Patterns

Being able to identify the user of a computer solely based on their typing patterns can lead to improvements in plagiarism detection, provide new opportunities for authentication, and enable novel guidance methods in tutoring systems. However, at the same time, if such identification is possible, new privacy and ethical concerns arise. In our work, we explore methods for identifying individuals from typing data captured by a programming environment as these individuals are learning to program. We compare the identification accuracy of automatically generated user profiles, ranging from the average amount of time that a user needs between keystrokes to the amount of time that it takes for the user to press specific pairs of keys, digraphs. We also explore the effect of data quantity and different acceptance thresholds on the identification accuracy, and analyze how the accuracy changes when identifying individuals across courses. Our results show that, while the identification accuracy varies depending on data quantity and the method, identification of users based on their programming data is possible. These results indicate that there is potential in using this method, for example, in identification of students taking exams, and that such data has privacy concerns that should be addressed.Peer reviewe

Ultrasonic Fouling Detector Powered by Machine Learning

Guided waves can be used to monitor structural health in industrial pipelines, and e.g. allow detection of accumulated precipitation on the surface of pipe. Propagation of guided waves in a tubular structure carrying possible fouling can be separated from a clean structure due to variation in wave propagation properties at the fouled area. In addition, multiple propagation paths around the tubular structure allow locating the fouled areas. In this study, we obtained dispersion curves of a tubular structure loaded with a local fouling layer of different thickness by using numerical simulations. We combined the dispersion curve information with simulated and measured times-of-arrival of guided wave propagation to second order helicoidal paths and used a Gaussian Process machine learning approach to estimate location of fouling on a steel pipe.Peer reviewe

Localizing a target inside an enclosed cylinder with a single chaotic cavity transducer augmented with supervised machine learning

Ultrasound is employed in, e.g., non-destructive testing and environmental sensing. Unfortunately, conventional single-element ultrasound probes have a limited acoustic aperture. To overcome this limitation, we employ a modern method to increase the field-of-view of a commercial transducer and to test the approach by localizing a target. In practice, we merge the transducer with a chaotic cavity to increase the effective aperture of the transducer. In conventional pulse-echo ultrasound signal analysis, location estimation is based on determining the time-of-flight with known propagation speed in the medium. In the present case, the dispersing field induces complexity to this inverse problem, also in 2D. To tackle this issue, we use a convolutional neural network-based machine learning approach to study the feasibility of employing one single chaotic cavity transducer to localize an object in 2D. We show that we indeed can localize an inclusion inside a water-filled cylinder. The localization accuracy is one diameter of the inclusion. The area that we can infer increases by 49% in comparison to using the same transducer without applying the proposed chaotic cavity method. (C) 2021 Author(s).Peer reviewe